NZ GREEN Grid project example:
Testing power demand: rf_38
Ben Anderson (b.anderson@soton.ac.uk, @dataknut)
Last run at: 2018-07-31 16:08:37
About
Report circulation:
- Public - this report is intended to accompany the data release.
License
Citation
If you wish to use any of the material from this report please cite as:
- Anderson, B. (2018) NZ GREEN Grid project example: Testing power demand: rf_38 Centre for Sustainability, University of Otago: Dunedin.
This work is (c) 2018 the University of Southampton.
Requirements:
This report uses the safe version of the grid spy 1 minute data which has been processed using https://github.com/dataknut/nzGREENGridDataR/blob/master/dataProcessing/gridSpy/processGridSpy1mData.R.
Support
This work was supported by:
- The University of Otago;
- The University of Southampton;
- The New Zealand Ministry of Business, Innovation and Employment (MBIE) through the NZ GREEN Grid project;
- SPATIALEC - a Marie Skłodowska-Curie Global Fellowship based at the University of Otago’s Centre for Sustainability (2017-2019) & the University of Southampton’s Sustainable Energy Research Group (2019-202).
We do not ‘support’ the code but if you have a problem check the issues on our repo and if it doesn’t already exist, open one. We might be able to fix it :-)
Introduction
This report provides summary analysis of one household as an example.
Load rf_38 data
The data used to generate this report is:
- /Volumes/hum-csafe/Research Projects/GREEN Grid/Clean_data/safe/gridSpy/1min/data/rf_38_all_1min_data.csv.gz
- /Volumes/hum-csafe/Research Projects/GREEN Grid/Clean_data/safe/survey/ggHouseholdAttributes.csv
## Parsed with column specification:
## cols(
## sample = col_character(),
## hhID = col_character(),
## newID = col_character(),
## Location = col_character(),
## nAdults = col_integer(),
## nChildren0_12 = col_integer(),
## nTeenagers13_18 = col_integer(),
## notes = col_character(),
## r_stopDate = col_date(format = "")
## )| sample | hhID | newID | Location | nAdults | nChildren0_12 | nTeenagers13_18 | notes | r_stopDate |
|---|---|---|---|---|---|---|---|---|
| Unison | rf_38 | rf_38 | Hawkes Bay | 2 | 2 | 0 | NA | NA |
Table @ref(tab:hhData) shows household attributes such as how many people live in this household.
Table @ref(tab:gsData) shows the initial grid spy 1 minute power data.
| hhID | dateTime_orig | TZ_orig | r_dateTime | circuit | powerW |
|---|---|---|---|---|---|
| rf_38 | 2015-03-25 16:51 | date NZ | 2015-03-25 03:51:00 | Heat Pump$4175 | 0.00 |
| rf_38 | 2015-03-25 16:51 | date NZ | 2015-03-25 03:51:00 | Lighting$4176 | 0.00 |
| rf_38 | 2015-03-25 16:51 | date NZ | 2015-03-25 03:51:00 | Incomer - Uncontrolled$4177 | 180.24 |
| rf_38 | 2015-03-25 16:51 | date NZ | 2015-03-25 03:51:00 | Hot Water - Controlled$4178 | 0.00 |
| rf_38 | 2015-03-25 16:51 | date NZ | 2015-03-25 03:51:00 | Kitchen, Dining & Office$4179 | 90.71 |
| rf_38 | 2015-03-25 16:51 | date NZ | 2015-03-25 03:51:00 | Laundry, Lounge, Garage, Bed$4180 | 61.46 |
Note that:
- the original dataTime (
dateTime_orig) and TZ (TZ_orig) have been retained so that the user can check for parsing errors (see https://github.com/dataknut/nzGREENGridDataR/issues/2) if required; - r_dateTime loads as UTC by default so we need to re-timezone it to NZ time before deriving any other times and dates.
# r_dateTime will have loaded as UTC (if you don't believe me, check :-)
gsDT <- gsDT[, r_dateTime := with_tz(r_dateTime, tzone = "Pacific/Auckland")]
t <- head(gsDT)
gsDT <- gsDT[, obsDate := lubridate::date(r_dateTime)]
gsDT <- gsDT[, obsTime := hms::as.hms(r_dateTime)]
knitr::kable(t, caption = paste0("Summary of grid spy data for", params$hhID, "tz corrected"))| hhID | dateTime_orig | TZ_orig | r_dateTime | circuit | powerW |
|---|---|---|---|---|---|
| rf_38 | 2015-03-25 16:51 | date NZ | 2015-03-25 16:51:00 | Heat Pump$4175 | 0.00 |
| rf_38 | 2015-03-25 16:51 | date NZ | 2015-03-25 16:51:00 | Lighting$4176 | 0.00 |
| rf_38 | 2015-03-25 16:51 | date NZ | 2015-03-25 16:51:00 | Incomer - Uncontrolled$4177 | 180.24 |
| rf_38 | 2015-03-25 16:51 | date NZ | 2015-03-25 16:51:00 | Hot Water - Controlled$4178 | 0.00 |
| rf_38 | 2015-03-25 16:51 | date NZ | 2015-03-25 16:51:00 | Kitchen, Dining & Office$4179 | 90.71 |
| rf_38 | 2015-03-25 16:51 | date NZ | 2015-03-25 16:51:00 | Laundry, Lounge, Garage, Bed$4180 | 61.46 |
gsDT <- gsDT[, month := lubridate::month(r_dateTime, label = TRUE)]
gsDT <- gsDT[, year := lubridate::year(r_dateTime)]Plot monthly power profiles
Figure @ref(fig:plotProfiles) shows plots overall mean power (W) per minute per month for each circuit to show:
- patterns of missing data (no lines)
- patterns of consumption
- possible dateTime issues (where consumption patterns seem to be stangely shifted in time)
- possible PV installation
Figure @ref(fig:plotProfiles) in rf_38 seems to indicate some potential time of day mis-alignments in late 2016. We suspect these may be due to incorrect date formats.
Figure @ref(fig:dateCheck) checks the original date format by month and year.
Figure @ref(fig:plotlyProfiles) repeats the original power demand analysis but just for the few ‘doubtful’ months so that we can check and also compare with another household where we do not see this (e.g. rf_06). It uses plotly to generate an interactive plot (html only).
| hhID | dateTime_orig | TZ_orig | r_dateTime | circuit | powerW | obsDate | obsTime | month | year |
|---|---|---|---|---|---|---|---|---|---|
| rf_38 | 2016-10-11 13:01 | date UTC | 2016-10-12 02:01:00 | Heat Pump$4175 | 0.00 | 2016-10-12 | 02:01:00 | Oct | 2016 |
| rf_38 | 2016-10-11 13:01 | date UTC | 2016-10-12 02:01:00 | Lighting$4176 | 123.39 | 2016-10-12 | 02:01:00 | Oct | 2016 |
| rf_38 | 2016-10-11 13:01 | date UTC | 2016-10-12 02:01:00 | Incomer - Uncontrolled$4177 | 198.64 | 2016-10-12 | 02:01:00 | Oct | 2016 |
| rf_38 | 2016-10-11 13:01 | date UTC | 2016-10-12 02:01:00 | Hot Water - Controlled$4178 | 0.00 | 2016-10-12 | 02:01:00 | Oct | 2016 |
| rf_38 | 2016-10-11 13:01 | date UTC | 2016-10-12 02:01:00 | Kitchen, Dining & Office$4179 | 0.00 | 2016-10-12 | 02:01:00 | Oct | 2016 |
| rf_38 | 2016-10-11 13:01 | date UTC | 2016-10-12 02:01:00 | Laundry, Lounge, Garage, Bed$4180 | 39.88 | 2016-10-12 | 02:01:00 | Oct | 2016 |
| rf_38 | 2016-10-11 13:02 | date UTC | 2016-10-12 02:02:00 | Heat Pump$4175 | 0.00 | 2016-10-12 | 02:02:00 | Oct | 2016 |
| rf_38 | 2016-10-11 13:02 | date UTC | 2016-10-12 02:02:00 | Lighting$4176 | 123.39 | 2016-10-12 | 02:02:00 | Oct | 2016 |
| rf_38 | 2016-10-11 13:02 | date UTC | 2016-10-12 02:02:00 | Incomer - Uncontrolled$4177 | 196.27 | 2016-10-12 | 02:02:00 | Oct | 2016 |
| rf_38 | 2016-10-11 13:02 | date UTC | 2016-10-12 02:02:00 | Hot Water - Controlled$4178 | 0.00 | 2016-10-12 | 02:02:00 | Oct | 2016 |
Figure @ref(fig:checkDoubtfulDates) repeats the earlier analysis but shows just the ‘doubtful’ months. If the plots do not show multiple date formats then it seems unlikely there is an issue.
Runtime
Analysis completed in 47.43 seconds ( 0.79 minutes) using knitr in RStudio with R version 3.5.0 (2018-04-23) running on x86_64-apple-darwin15.6.0.
R environment
R packages used:
- base R - for the basics (R Core Team 2016)
- data.table - for fast (big) data handling (Dowle et al. 2015)
- lubridate - date manipulation (Grolemund and Wickham 2011)
- ggplot2 - for slick graphics (Wickham 2009)
- readr - for csv reading/writing (Wickham, Hester, and Francois 2016)
- knitr - to create this document & neat tables (Xie 2016)
- nzGREENGridDataR - for local NZ GREEN Grid project utilities
Session info:
## R version 3.5.0 (2018-04-23)
## Platform: x86_64-apple-darwin15.6.0 (64-bit)
## Running under: macOS High Sierra 10.13.6
##
## Matrix products: default
## BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib
##
## locale:
## [1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] hms_0.4.2 rmarkdown_1.10 knitr_1.20 readr_1.1.1
## [5] plotly_4.8.0 ggplot2_3.0.0 lubridate_1.7.4 data.table_1.11.4
## [9] nzGREENGrid_0.1.0
##
## loaded via a namespace (and not attached):
## [1] tidyselect_0.2.4 purrr_0.2.5 reshape2_1.4.3
## [4] lattice_0.20-35 colorspace_1.3-2 htmltools_0.3.6
## [7] viridisLite_0.3.0 yaml_2.2.0 rlang_0.2.1
## [10] later_0.7.3 pillar_1.3.0 glue_1.3.0
## [13] withr_2.1.2 sp_1.3-1 bindrcpp_0.2.2
## [16] jpeg_0.1-8 bindr_0.1.1 plyr_1.8.4
## [19] stringr_1.3.1 munsell_0.5.0 gtable_0.2.0
## [22] htmlwidgets_1.2 RgoogleMaps_1.4.2 mapproj_1.2.6
## [25] evaluate_0.11 labeling_0.3 httpuv_1.4.5
## [28] crosstalk_1.0.0 Cairo_1.5-9 highr_0.7
## [31] proto_1.0.0 Rcpp_0.12.18 xtable_1.8-2
## [34] geosphere_1.5-7 promises_1.0.1 openssl_1.0.2
## [37] scales_0.5.0 backports_1.1.2 jsonlite_1.5
## [40] mime_0.5 rjson_0.2.20 png_0.1-7
## [43] digest_0.6.15 stringi_1.2.4 shiny_1.1.0
## [46] dplyr_0.7.6 grid_3.5.0 rprojroot_1.3-2
## [49] tools_3.5.0 magrittr_1.5 maps_3.3.0
## [52] lazyeval_0.2.1 tibble_1.4.2 crayon_1.3.4
## [55] tidyr_0.8.1 pkgconfig_2.0.1 assertthat_0.2.0
## [58] httr_1.3.1 rstudioapi_0.7 R6_2.2.2
## [61] ggmap_2.6.1 compiler_3.5.0References
Dowle, M, A Srinivasan, T Short, S Lianoglou with contributions from R Saporta, and E Antonyan. 2015. Data.table: Extension of Data.frame. https://CRAN.R-project.org/package=data.table.
Grolemund, Garrett, and Hadley Wickham. 2011. “Dates and Times Made Easy with lubridate.” Journal of Statistical Software 40 (3): 1–25. http://www.jstatsoft.org/v40/i03/.
R Core Team. 2016. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.
Wickham, Hadley. 2009. Ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. http://ggplot2.org.
Wickham, Hadley, Jim Hester, and Romain Francois. 2016. Readr: Read Tabular Data. https://CRAN.R-project.org/package=readr.
Xie, Yihui. 2016. Knitr: A General-Purpose Package for Dynamic Report Generation in R. https://CRAN.R-project.org/package=knitr.